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This investigation had two major purposes. The first 
was to explore the use of an inferential technique called Tukey's 
Jackknife in establishing a confidence interval about cooefficient 
alpha reliability. The second purpose was to study the robustness of 
the Feldt and the jackknife procedures when the data fails to satisfy 
usual normality assumptions. Osing a linear model of test score, 
computer simulated data representing a matched item-examinee sample 
from th3ir respective populations was employed. The two jackknife 
procedures, is this situation, were less conservative. For log^normal 
distributions of examinee score components, Feldt^s procedure gave 
reasonable interval estimates for low values (=.60) of population 
alpha, but for higher values of population alpha (=.90) , the Feldt 
procedure was not found to be robust. The two Jackknife procedures, 
on the other hand, were found to be relatively robust over the entire 
range of investigation of values of alpha. None of the procedures was 
found useful in interval estimation of coefficient alpha when the 
distribution of examinee score components were of extreme 
double-exponential form. (RC) 
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A COMPARISON OF lOTERVAL ESTIMATION OF COEFFICIENT ALPHA 
USING THE FELDT AND THE JACKKNIFE PROCEDURES 

1 

Tej N. Pandey and Lawrence J. Hubert 

University of Wisconsin, Madison BEST COPlt AVAlU&fc 

INTRODUCTION 

The ca'j culated reliability of a test based on a small sample of subjects 
is an estimate of the population reliability, and hence, is subject to sampling 
fluctuation. Until quite recently little application has been made of statis- 
tical inference techniques to the reliability coefficient. Using analysis of 
variance framework, Ebel (1951) and Jackson and Ferguson (19^1) made the first 
attempts to relate reliability coefficient estimates to the well known 
F-distribution. However, Kristoff (1963, 1970) and Feldt (1965) presented 
complete sampling theory of reliability estimates and also methods to apply it. 

Kristoff derived the sampling distribution of a maximum likelihood estimate 

of the population Value of alpha using the technique of transformation of 

variables. The sampling theory derived by Feldt, which is also of concern 

here, was based on normality assumptions regarding the true and error score 

distributions. To be more precise, consider the following components of 

variance model employed by Feldt, 

x^. = + + aj + e^^j; i = l,...,n; j = l,...,k, 

and |Ltis some overall constant, t N(o ,G^t^) , jaj/v N(o, ^r'a^)], N (o,C7^e^), 

and the n+k+nk random variables { > f^j^> ^'^^ mutually independent. 

Under these constraints the usual F-ratio provides a suitable test of Ho: ^t - To " 
against Hi:'j"t ^<»b ; however, whenever tj has a positive kxirtosis, Scheffe'' (1959) 
has shown that the true significance level of the proposed F-test is higher than 
the assumed value of the significance level; and frequently, it is substantially 
higher. 
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It is, quite conceivable that real data commonly found in educational and 
social sciences does not always meet the rigorous assumption of normality. 
Consequently, Feldt's statistic (M) may have limited applicability if it 
fails to be robust to deviations from normality. 

The present investigation had two major purposes. The first was to explore 
the use of a relatively new inferential technique called Tukey's Jackknife 
(Miller, 196^), in establishing a confidence interval about 0(r. The second 
purpose was to study the robustness of the Feldt and the jackknife procedures 
when the data fails to satisfy usual normality assumptions. Besides the claims 
of being a competitor of the usual normal theory F-test (see Arvesen, 1969)1 
the jackknife procedure has been shown in many situations to behave robustly 
against non-normality. 

The jackknife Procedure 

The jackknife procedure used in obtaining an interval estimate of a para- 
meter is based upon Quenouille's (19^9 ^ 1956) work on reduction of bias in an 
estimator. Tukey's contribution is in the extention of Quenouille's procedure 
as an infercutial technique that may be appropriate when either the distri- 
butional assumptions are in question or distribution theory is impossible to 
derive. The name "jackknife" procedure due to Tukey naturally suggests that 
it is a generally applicable tool like the boy scout's jackknife, though many 
of its jobs could be better accomplished using specialized tools if such tools 
were available. However, Miller (196^, I968) has shown that Tukey 's jackknife 
is a valid mathematical technique for constructing confidence intervails. 

The jackknife procedure can be defined as dividing a set of observations 
into a set of mutually exclusive and exhaustive groups, obtaining estimates from 
combination of these groups, and finally averaging these estimates. To be more 
precise, let Xl^, • . . , Xj^ be N independent observations, identically 
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distributed random variables, having an unknown parameter 0 of the ccmraon 
density function Fq. Furthermore, we assume that a method for estimating 0 
is available.. In the jackknife procedure, N observations are first grouped 
into t groups' of ^ observations each such that N=tn, i.e., the t groups are 
as follows: 

As a notational conventions, let 0^ be some estimate of the parameter 0 

A 

based on all t groupr, and let Qj^ be the estimate of 0 baf?ed upon the deletion 
or the i-th group of observations, i.e., based upon (K n) observations. New 
estim^res of 0, called pseudo-values, are formed by taking the following 
combination of O^q and : 

9*i = t G.o - - 1) e.i for i = l,...,t. 
Tli^ jackknife estimate of 0 is the mean of the pseudo-values 

e*. " 1 = 1 "T" 

An n^stimnte of the standard error of tho jackknife estimate Is p:ivon by: 

1/2 



Li=l t(t - 1) J 



The jackknife estimate of 0 possesses the interesting property that if 

A 2 
0 is biased of the order l/N, then ©le. reduces the bias to the order l/N . 

Moreover, Tukey (1958) suggested that in many situations the t pseudo-values 

(^4-1 ' could be treated as t approximately independent, identically 

distributed observations from which an approximate confidence interval of 0 

could be constructed from the student - t distribution. Tukey 's proposal 

implies that the quantity 




is approximately distributed as a ^tudent - t with t - 1 degrees of freedom. 
Thus a 100r% confidence interval for 0 is 

O -3- ' 
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Jackkni.f.ing Tranaformation of Statistics 

Miller (196^) in discussing Tukey's conjucture, actually proved that at 

leafit for the transformation of means, the pseudo-values are asymptolically 

distributed normally. However, in the jackknife estimate a slightly stronger 

assumption of bounded second derivative near the origin was required as compared 

to the requirement of first derivative near the origin for the unjackknifed 

estimate of the transformation of means. 

In a subsequent paper, Miller (1968) extended his discussion of the mean 
to the case where G is the sample variance, or a transformation of the sample 
variance. For a moderate sample size, his results have indicated that the 
jackknife t-test is a valid competitor to the F-test if the data are normal, 
and moreover, the jackknife gives almost correct significance levels if the 
data are not normal, unlike the F-test. 

Miller's theorems on the jackknifing of means and variances were extended 
to the consideration of U-statistics, or a function of several U-statistics by 
Arvesen (I969)» U-statistics embrace a large class of statistics including 
the sample mean, variance, and estimates of variance components in ANOVA models. 

Application of jackknife for Interval Estimation 

It has been pointed out by Mosteller and Tukey (1968) that the jackknife 
technique can be applied to the same data base in a variety of ways. This 
variety van occur in two ways. One of these is the particular function of the 
estimator to be jackkJiifed. For e^^ample, one can jackknife log 0 or 0 instead 



of 0. There may be sone advantage in jackknifing one expression rather than 
other, or one function can lead towards desirable results whereas the other 
function may not be useful at all. Rogers (l97l) while jackknifing disattenuated 



A 



correlation coefficient 




found that performance of the jackknife 



procedure on the statistic 



7/s- 



was slightly superior to the performance 
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of the jackknifo f)rocodure on r(T , T ) in certain nituatious. The transformations 

X y 

reforred to here are known as variance stabilizing transformations. Arvesen (19^9) 
has shovm that log transformation is a variance. Stabilizing transformation of 
the ratio of variance component estimates in a two-way Model II ANOVA. Arvesen 
confirmed empirically that it was more useful to jackknife log of ratio of 
vnrianco comytonent er>timates as comyiared to simple ratio of component estimates 
in AfJOVA l() ')l)Lriiri [ritcrvnl ''\';t i.rn'it'-'rj . The f)robl.^^m, hov/ov^r, in thr\\. for* many 
uneful tir^hics v-'jriance stabilizing transformations are unknown. 

Th*-^ other way riu^^ to which variety in jackknifing occurs is the choice of 
forming sub-grouf)S. V/hen the data is in the form of a vector, Miller (lOCh) 
suggested to keep the number of observations (N) equal to the number of groups 
(t) , resulting in n~l, i.e., one observation per group. For data in the form 
of a matrix, Cronbach, Rajaratnam, Gleser, and Nanda (1972) as well as Collins 
(1970) used as many groups as the number of observations in the :.iatrix. This 
investigation uses the foregoing as well as other procedure of group formation. 

Method 

To 'ietermine- v/hlch particul'jr statictic would "poli:jh up" the behavior 
of the jackknife technique in giving interval estimates of coefficient alpha, 
the following five functions were jackknifed. It has been shown by Pandey (1973) 

that each of these statistics is a U-statistic: 

1. 5=1- MSp/MSn^r; A. §t= MSp/MSp^u 

2. logja) = logjl - MSp/MSp^y} 5. log^(9,) = log^CMSp/MSp^u^ , 

3. = k logj (H-a)/(l -5)' 
^ a ^ 

v/here jC - estimate of coefficient >7C MSp and MSpxu are the mean squares for 
persons and persons by units in a two-way Model II analysis of variance. 

Regarding choice of functions to be jackknifed, choice of is obvious 
because it is a natural function to be jackknifed. Loge was chosen because 



log transformation is known to be a variance stabilizing transformation for 

sample variance. 0^ and loge f^t were used by Arvesen and Schmitz (1970) and 

•ij> was intuitively considered to be a good transformation because it is a 

2 

variance stabilizing transformation for sample correlation coefficients. 

For each of the statistic jackknifed, two methods of group formation were 
ufied. These are the ''technique of eliminating both rows and columns" and the 
"tochniqup of eliminating rows only'' computation of pseudo-values and their 
variance are given in Pandey (1975) • 



DATA AND EXPERIMENTAL DESIGN 
The continuously scored data representing a matched item-examinee sample 
from their respective populations was simulated on UNIVAC 1108 computer, the 
characteristics of which could be manipulated as desired by the experimenter. 
The following model representing a two-dime neional array of observations of 
size nxk was used to define the test scores: 

In this model |Lc is an arbitrary constant, t^ is the effect associated with 
examinee i, a. is the effect associated with item i, and e. . is a random error 
corresponding to a particular observation x. .. The size of the array, value 
of 1^ , nature of the distributions of a^. and ^j^y sind associated parameters 
were kept constant throughout the experiments, whereas the reliability of the 
fixed length test and the distributional form of the examinee effect t^ were 
manipulated systematically. Using three distributional forms of t^ (Normal, 
Log-normal, and Double-exponential) and three values of population coefficient 
alpha (.60, ,751 and .90) resulted in nine simulation experiments. Each of 

Lord (197^0 has recently shown that Zj^ is the correct variance stabilizing 
transformation of the stepped-up reliability coefficient. 
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the experiments was performed using 1000 replications; the number of times 
the computed confidence interval did not enclose the population value of 
coefficient alpha, and the mean interval length were recorded for the Feldt 
and the ten jackknife procedures at three values of confidence coefficients 
(.90, .95, .99). 'Various methods were compared with respect to their empirical 
significance values, as well as the tightness of the confidence bounds. The 
tables in the Appendix provide in part the results of the simulation experiments. 

RESULTS AM) DISCUSSION 

The jackknife procedure using technique of elimination of row only, 
involving Fisher's z-transformation on alpha and log(MSp/MSp^y.) were found to 
have potential for interval estimation of coefficient alpha for the normal and 
log-normal distributions of examinee score components, the Feldt procedure was 
found to give slightly conservative estimates of alpha, consistent with the 
earlier results of Feldt (1965)* The two jackknife procedures, in this situation 
were less conservative. For log-normal distribution of examinee score components, 
Feldt *s procedure gave reasonable interval estimates for low values (=.60) of 
population alpha, but for higher values of population alpha (=.90), the Feldt 
procedure was not found to be robust. The two jackknife procedures, on the 
other hand, were found to be relatively robust over the entire range of 
investigation of values of alpha. None of the procedures was found useful in 
interval estimation of coefficient alpha when the distribution of examinee 
score components were of extreme double -exponential form. 

Collins (1970) used the jackknife technique to study generalizability 
coefficients. Based on his results, he recommended against using the jackknife 
as an inferential technique for generalizability coefficients. The results of 
the present investigation are similar to those of Collins for the technique 
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of row and column elimination. However, the techrxique of row elimination only 
(not studied by Collins) proved useful in constructing confidence interv.'il 
for coefficient alpha. Since the technique was found relatively robust, it 
is recommended that jackknife interval estimates of coefficient alpha be used 
in computerized item analysis packages. 
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Table I 

Results of Feldt's Normal Theory Procedure 



Popu- Nominal Confidence Coefficient 

iation 

Alpha .99 ^95 ,90 

Lower Upper Error Ave, Lower Upper Error Ave. Lower Uppe'- Error Ave. 
Tail Tail Freq, Inter- Tail Tail Freq. Inter- Tall Tall Freq. Inter 
val val val 



1 


N 


.60 


6 


0 


6 


.6893 


26 


27 


53 


.5176 


54 


52 


106 


.4321 


2 


N 


.75 


6 


6 


12 


.4324 


23 


24 


47 


,3z47 


54 


50 


104 


. 2710 


3 


N 


.90 


2 


3 


5 


.1724 


23 


22 


45 


.1295 


52 


63 


115 


.1081 


4 


LN 


.60 


8 


0 


8 


.6910 


30 


26 


56 


.5190 


56 


48 


104 


.4332 


5 


LN 


.75 


8 


8 


16 


.4358 


27 


27 


• 54 


.3273 


56 


58 


114 


.2732 
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LN 


.90 


15 


11 


26 


.1779 


37 


38 


75 


.1336 


64 


77 


141 


.1115 
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E 


.60 


1 


0 


1 


.9708 


103 


107 


210 


7290 


127 


211 


338 


.6086 


8 


E 


.75 


40 


110 


150 


.7121 


121 


237 


358 




114 


362 


476 


.4464 


9 


E 


.90 


197 


115 


312 


.3371 


303 


210 


513 


.2532 


203 


411 


614 


.2113 
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No. 
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Table % 

Results on Row Only Elimination Procedure: Function Jackknifed z* 



Slrau- Dlstrl- Nominal Confidence Coefficient 

lation , ^. " lation 

No. Alpha ^9 ^95 

Lower Upper Error Ave. Lower Upper Error Ave. Lower Upper Error Ave. 
Tail Tail Freq. Inter- Tail Tall Freq. Inter- Tail Tail Freq. Inter- 
val val val 
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N 


.60 


2 


5 


7 


.7047 


21 


30 


51 


.5248 


50 


45 


95 


.4362 
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N 


.75 


4 


3 


7 


.4759 


17 


21 


38 


.3458 


48 


48 


96 


.2845 
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N 


.90 


2 


7 


9 


.1988 


14 


36 


50 


.1404 


42 


52 


94 


.1144 


4 


LN 


.60 


3 


8 


11 


.7094 


25 


37 


62 


.5282 


51 


56 


107 


.4390 


5 


LN 


.75 


4 


8 


12 


.4880 


20 


23 


43 


.3541 


43 


51 


94 


.2913 


6 


LN 


.90 


1 


8 


9 


.2228 


17 


38 


55 


.1555 


40 


74 


114 


.1261 


7 


E 
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81 


114 


195 


.8207 


106 


143 


249 


.6224 


132 


180 


312 


.5210 


8 


£ 


.75 


36 


83 


119 


.7858 


93 


165 


258 


.5859 


111 


232 


343 


.4867 
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E 


.90 


78 


123 


201 


.4702 


129 


236 


369 


.3325 


197 


269 


466 


.2704 
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